For the Seqera Platform set up (easier), just follow the batch-forge instructions from seqera
These instructions are mainly based on Luke Pembleton's blog and discussions with Eva Price.
1.1- Create an IAM user (add programmatic access at later stage in step 5 below- not like in blogs). Don’t bother about creating the group and adding policies now, just create the user for now
1.2- Create a user group and add the previous user to it.
1.3- Go to user group and create inline polices as per the NextFlow website (https://www.nextflow.io/docs/latest/aws.html). I did this one by one.
Notes:
batch:TagResource (as in my first attempt my jobs were failing with nf_user is not authorized to perform: batch:TagResource), and CloudWatch Logs:GetLogEvents, as someone also suggested to add this and I could see some warnings in my .nextflow.log files about itOnce all the permissions are added, click next and save the policy
1.4- Create a role to be able to use spot instances -> Roles -> Create Role -> AWS service, Use Case = EC2 -> AmazonEC2SpotFleetTaggingRole. Then click next until is created (didn’t need to add permission, just to add a name for the Role at the end)
1.5- Create access keys to give programmatic access to the user: User -> Security credentials -> Create access key -> Command Line Interface -> Create access keys
Amazon ECS is a container orchestration service that enables you to run and manage Docker containers on a cluster of EC2 instances. When you create an ECS cluster that uses EC2 instances, you need to specify an AMI for those instances. The AMIs are templates to create EC2 instances, as they contain the software needed to launch the instances (OS, apps, etc). These AMI can be custom AMIs or ECS-optimized AMIs, which are optimized for running containers on ECS. In summary:
2.1- EC2 dashboard -> Instances -> Launch instance -> Browse more AMIs. Then Click AWS Marketplace AMIs -> search for ECS -> filter by “Free”. Select one of the recent ones -> click on Subscribe now
2.2- Select instance type = t2.micro, and In Key pair -> create new key pair (save the *.pem file)
2.3a- Check that in the network settings, “Auto-assign Public ID” is ENABLED, otherwise, click “Edit” and enable it (this happens if for example I used the default VPC from UCL-AWS, and not do step 0 – create a VPC)
2.3b- I do need to create a new security group that allows ssh traffic, otherwise I will not be able to ssh to the instance. Configure storage = 30Gb gp3 (free tier). Then click “Create Instance”
2.4- Check that the instance has aws-cli, otherwise you need to install it as described in Appendix B
2.5- Under the Instances menu select your relevant instance and click Actions -> Images and Templates -> Create Image. Take note of the AIM ID (not the name)
2.6- Once the AMI has been created, the instance can be stopped or terminated, as this instance was used solely for the purpose of creating the AMI
NOTE1: The point of this step is to get one of the existing AMIs, and to build up on that by adding the aws-cli to allow data movement to and from S3 buckets. But if the AMI I’m using (with Amazon Linux 2023) already has aws-cli installed, I guess there is no point of doing this? I am basically taking an image and making an image of the same image without having added anything to it.. :?
NOTE2: I'm also confused about the aws path given in the nextflow.config. If I actually give the actual aws path from the AMI, my jobs fail. I needed to add the default /home/ec2-user/miniconda/bin/aws even though my AMIs never used conda to install aws, either they came with it, or I did it as in APPENDIX B'
3.1- AWS Batch dashboard -> compute environments -> create -> EC2
3.2- Select “managed”. If this is the first time setting up a batch environment, AWS will create the relevant service role and instance role. Just ensure “Create new Role” is selected. Alternatively, select “AWSServiceRoleForBatch” service role, and “ecsInstanceRole” as Instance role -> Next
3.3- Click “Use EC2 Spot instances”. In Spot Flee Role, reload and choose the EC2SpotFleeTaggingRole that was created earlier in step 1.4. Leave 0 CPUs for minimum and desired, and whatever number of CPUs as max. Leave default “optimal” as Allowed Instance Types.
NOTE1: I originally was doing this on my personal account with the free tier, so I had this as min=0, max=4, and noticed that when I did my hello world tests, I had some m4.xlarge instances running! I changed this to min=0, max=1, but the next day had a Zero-spend budget exceeded email as that costed me £0.02. Be careful with this if you want to stay within the free tier.
NOTE2: if you want to stay within the free tier, and select MAX=1, then your jobs will stay in RUNNABLE forever. This is most likely because there are no instances wioth 1 CPU available via spot.
3.4- Under additional configuration, select the key pair created earlier in step 2.2. Choose “BEST FIT” as allocation strategy. Under “EC2 configuration” click Add EC2 configuration, select Amazon Linux 2023 as image type, and type the AMI ID created earlier in the “Image ID override” box -> Next
3.5- In Network configuration leave defaults as I’m not sure what to do here -> Create compute environment
3.6- In the other blog they add an additional step. I need to add this step because if I don’t do this, my jobs die with:
“Download failed: s3://lconde-nf-bucket/nf-wd/03/96ed2a8b6170edfb4d9708a330f9dc/.command.run to - An error occurred (403) when calling the HeadObject operation: Forbidden; upload failed: ./.command.log to s3://lconde-nf-bucket/nf-wd/03/96ed2a8b6170edfb4d9708a330f9dc/.command.log An error occurred (AccessDenied) when calling the PutObject operation: Access Denied”
Update the ecsInstanceRole
AWS Batch just created the ecsInstanceRole for us but we need to add the S3 role to allow access to data stored in S3 buckets. Go to the IAM service as done in Step 6 in the Setting up a programmatic user with IAM section. Under Roles enter
ecsin the search box and click on the ecsInstanceRole. Click the Attach policies and enterS3in the filter box and select the AmazonS3FullAccess, then click Attach policy.You should have both the AmazonS3FullAccess and the AmazonEC2ContainerServiceforEC2Role policies attached to this role.
3.7- Still within Batch AWS dashboard -> Job queues -> Create. Choose Amazon EC2 -> name the queue -> connect to compute environment created earlier -> create job queue
4.1- Create a new private bucket in relevant region. I just added a name and left the rest with default values
4.2- Go to the bucket, and create a new folder within the bucket to serve as the Nextflow working directory (take note of the S3 URI address)
process {
cpus = 1
memory = '6.GB'
time = '1.h'
}
params {
max_cpus = 1
max_memory = '6.GB'
max_time = '1.h'
}
profiles {
local {
process.executor = 'local'
singularity.enabled = true
singularity.autoMounts = true
}
myriad {
process.executor = 'sge'
process.penv = "smp"
process.clusterOptions = { "-l mem=${task.memory.giga}G" }
singularity.enabled = true
singularity.autoMounts = true
}
aws {
process.executor = 'awsbatch'
process.queue = 'nf-queue'
process.container = 'ubuntu'
docker.enabled = true
aws {
batch {
cliPath = '/home/ec2-user/miniconda/bin/aws'
// region = 'eu-west-2'
// jobQueue = 'nf-queue'
}
// client {
// uploadMaxRetries = 5
// }
region = 'eu-west-2'
}
wave {
enabled = true
// containerConfig {
// from = 'Dockerfile'
// }
}
fusion {
enabled = true
}
workDir = 's3://lconde-nf-bucket/nf-wd/'
//params.outdir = 's3://lconde-nf-bucket/output'
}
}
module load blic-modules nextflow
source ~/.bash_profile # to reset tower credentials
nextflow run lconde-ucl/hello_2 -r main --outdir results_hello -with-tower -profile local
nextflow run lconde-ucl/hello_2 -r main --outdir results_hello -with-tower -profile myriad
nextflow run lconde-ucl/hello_2 -r main --outdir results_hello -with-tower -profile aws
nextflow run lconde-ucl/hello_2 -r main --outdir s3://lconde-nf-bucket/results_hello -with-tower -profile aws
nextflow run nf-core/demo --input ./param_files/samplesheet_demo.csv --outdir results_demo -with-tower -profile local
nextflow run nf-core/demo --input ./param_files/samplesheet_demo.csv --outdir results_demo -with-tower -profile myriad
nextflow run nf-core/demo --input ./param_files/samplesheet_demo.csv --outdir results_demo -with-tower -profile aws
Creating a public subnet in AWS involves a series of steps within the AWS Management Console. Here's a detailed guide on how to do it:
10.0.1.0/24).0.0.0.0/0 (this is the route for all IPv4 traffic).
To SSH into your EC2 instance on AWS, you'll need the public IP address of your instance (find it in the AWS console -> instance) and the key pair file (.pem) associated with it with permissions set to allow only the owner to read and write (chmod 400 /path/to/your-key.pem). The username depends on the operating system of your EC2 instance. For Amazon Linux, the default username is ec2-user.
ssh -i nf_key_pair.pem [ec2-user@](mailto:ec2-user@18.175.122.220)18.130.78.47
sudo yum update
sudo dnf upgrade --releasever=2023.4.20240528
The version of amazon linux I have here already comes with aws.
In a previous instance I had created, aws was not available so I had to install it. I tried following the instructions from the blog but it did not work for me, as conda uses too much memory for my t2.micro instance. So installed aws-cli as described in https://docs.aws.amazon.com/cli/v1/userguide/install-linux.html. It needs python3.8+ that can be installed as described in https://stackoverflow.com/questions/61993914/how-do-i-install-and-use-python-3-8-on-an-amazon-ec2-linux-instance
sudo amazon-linux-extras install python3.8
sudo yum install -y unzip nano
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install
aws –version